CNTRLPLANE-3523: Retry on InvalidRouteTableId.NotFound during route creation#8747
CNTRLPLANE-3523: Retry on InvalidRouteTableId.NotFound during route creation#8747vismishr wants to merge 1 commit into
Conversation
AWS eventual consistency can cause CreateRoute to fail with InvalidRouteTableId.NotFound immediately after CreateRouteTable returns. Add retry with backoff for this error, matching the existing pattern in CreateVPCS3Endpoint.
|
Pipeline controller notification For optional jobs, comment This repository is configured in: LGTM mode |
|
Skipping CI for Draft Pull Request. |
|
@vismishr: This pull request references CNTRLPLANE-3523 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "5.0.0" version, but no target version was set. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository YAML (base), Central YAML (inherited) Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughIn 🚥 Pre-merge checks | ✅ 11✅ Passed checks (11 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: vismishr The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #8747 +/- ##
==========================================
- Coverage 41.79% 41.79% -0.01%
==========================================
Files 759 759
Lines 94037 94047 +10
==========================================
Hits 39304 39304
- Misses 51983 51993 +10
Partials 2750 2750
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
|
@vismishr: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
Now I have all the evidence. Let me compile the final report: Test Failure Analysis CompleteJob Information
Test Failure AnalysisErrorSummaryBoth codecov checks fail because the PR adds 15 new/modified lines to Root CauseThe root cause is straightforward: the modified file Specifically:
The codecov.yml configuration has no explicit Critically, these checks are not required for merge. The repository's branch protection does not list Recommendations
Evidence
|
What this PR does / why we need it:
hcp create cluster awsfails withInvalidRouteTableID.NotFounddue to a race condition with AWS eventual consistency. The CLI creates a route table and immediately tries to add routes before AWS has propagated the resource. This results in a 100% failure rate with no retry logic to handle the transient error.This PR adds retry with exponential backoff on
InvalidRouteTableId.NotFounderrors when creating routes after route table creation, matching the existing pattern already used inCreateVPCS3Endpoint.Files changed:
cmd/infra/aws/ec2.go—CreatePrivateRouteTableandCreatePublicRouteTableChanges:
CreatePrivateRouteTable: AddedinvalidRouteTableIDto the existing retriable error check alongsideinvalidNATGatewayErrorCreatePublicRouteTable: Wrapped theCreateRoutecall inretry.OnErrorwith backoff oninvalidRouteTableIDBoth use the existing
retryBackoff(5 steps, 3s base, 3x factor) already defined and proven in production viaCreateVPCS3Endpoint.Which issue(s) this PR fixes:
Fixes https://redhat.atlassian.net/browse/CNTRLPLANE-3523
Special notes for your reviewer:
CreateRoutewith the same destination CIDR on the same route table either succeeds or returnsRouteAlreadyExists.InvalidRouteTableId.NotFound— any other error fails immediately.invalidRouteTableIDwithretryBackoff) is already used inCreateVPCS3Endpointin the same file.Checklist:
Summary by CodeRabbit
Release Notes